Internet Info 1994 March/Internet Info CD-ROM (Walnut Creek) (March 1994).iso/standards/CCITT/1992/X/x290_2.asc

Text File | 1993-07-15 | 19.9 KB | 476 lines

7.Test methods

7.1    Introduction

       The testing of a given OSI* protocol can require the use of several test
methods, as systems under test can come in several configurations, and vary in
terms of their ability to allow ways of producing effects applicable to a layer
boundary.

       This section first characterizes the features of the system  under  test
which are to be taken into consideration, next defines the possible test
methods in abstract terms, and finally provides guidance on their applicability to
real systems.

7.2    Classification of real open systems and IUTs for conformance testing

7.2.1  Classification of systems under test

7.2.1.1There is a relation between the test methods and the configurations of  the
real open systems to be tested. The appropriate test methods vary according to:

       a)   the main function of the system (end-system or relay-system);

       b)   which layers use OSI* protocols;

       c)   whether the alternative of non-OSI* protocols is also available.

7.2.1.2The following configurations of systems have been identified for the purposes
of conformance testing, as illustrated in Figures 2 to 4.  Configurations 1 to 3 are
the basic configurations of systems under test (SUTs):

       a)   Configuration 1: 7-layer open system (end-system)

                        These systems use OSI* Recommendation* protocols in all 7 layers.

       b)   Configuration 2: Partial (N)-open system (end-system)

                        These systems use OSI* Recommendation* protocols in layers 1 to N.

       c)   Configuration 3: Open relay-systems

            These use OSI* protocols in layers 1 to 3 (Network relay-systems)  or 1
            to 7 (Application relay-systems).

7.2.1.3Other configurations can be derived from the basic configurations.

       A SUT can be a combination of basic configurations 1 and 2, allowing the
alternative of using OSI* and non-OSI* protocols above layer N (see Figure 5).












7.2.2  Identification of the implementation under test (IUT)

       An implementation under test (IUT) is that part of a real open system which
is to be studied by conformance testing. It should be an implementation of one or
more adjacent OSI* protocols.

       IUTs can be defined for configurations 1 and 2 of SUTs as single-layer IUTs
(one single layer of the SUT is to be tested), or as multi-layer IUTs (a set of any
number of adjacent layers of the SUT to be tested in combination).

       An IUT defined in an open relay-system will include at least  the  layer
which provides the relay function.

       When OSI* and non-OSI* protocols exist in a system, the IUT(s)  will  be
defined for the OSI* mode(s) of operation. Testing non-OSI* protocols is outside the
scope of this Recommendation.

       Clients and test laboratories will agree on what part of the SUT will be
considered to be the IUT.










7.3    Abstract testing methodology - general

       Test methods need to refer to an abstract testing methodology, based upon
the OSI reference model. Considering first end-systems (7-layer or partial (N)-open
systems) and single layer IUTs within these systems, abstract test methods  are
described in terms of what outputs from the IUT are observed and what inputs to it
can be controlled. More specifically, an abstract test method is  described  by
identifying the points closest to the IUT at which control and observation are to
be exercized.

       The OSI* protocol Recommendations* define allowed behaviour of a protocol
entity (i.e., the dynamic conformance requirements) in terms of the protocol data
units (PDUs) and the abstract service primitives (ASPs) both above and below that
entity. Thus the behaviour of an (N)-entity is defined in terms of the (N)-ASPs and
(N-1)-ASPs (the latter including the (N)-PDUs).

       If an IUT comprises more than one protocol entity, the required behaviour
can be defined in terms of the ASPs above and below the IUT, including the PDUs of
the protocols in the IUT.

       The starting point for developing test methods is the conceptual testing
architecture, illustrated in Figure 6.  It  is  a  "black-box"  active  testing
architecture, based on the definition of behaviour required of the IUT.

       The action of the tester shown in Figure 6 can either be applied locally,
in which case there is a direct coupling  within  the  system  under  test,  or
externally via a link or network. The two sets of interactions, above and below the
IUT, can, in practice, be observed and controlled from several different points,
locally or externally.







       The possible points of control and observation (PCOs) are identified  by
three factors:

       a)   whether it is the ASPs or PDUs which are observed and controlled;

       b)   the layer identity of the ASPs or PDUs concerned;

       c)   whether they are controlled and observed within the system under test
            or in a system remote from the system under test - if the latter then
            the ASPs are  distinguished  by  the  addition  of  a  double-quote
            character(").

       Possible PCOs within the SUT are illustrated in Figure 7(a). Possible PCOs,
when the activity below the IUT  is  controlled  and  observed  externally  are
illustrated in Figure 7(b). It can be seen from these figures that there  is  a
multiplicity of possible PCOs in different layers, which offer different degrees of
control and observation of IUT behaviour. This Recommendation makes a selection
from this set of possible PCOs, defining a limited number of abstract test methods.


       If control and observation below, or external from, the IUT is specified in
terms of ASPs, it will include control and observation of the PDUs carried by those
ASPs; but if it is specified in terms of PDUs (at layer N) then the ASPs (at layer
N-1) are not considered to be controlled or observed.

       It is assumed that the ASP activity  below  the  IUT  can  at  least  be
observable and controllable via the peer activity in a remote testing system  -
i.e., the corresponding ASP"s. Thus  when  the  ASPs  below  the  IUT  are  not
controllable nor observable locally, conformance testing  can  be  carried  out
externally, provided that the underlying service offered is sufficiently reliable
for control and observation to take place remotely.

       It is possible that  the  ASP  activity  above  the  IUT  might  not  be
controllable nor observable, in which case this activity is said to be hidden.

       SUTs are not required to provide access to layer boundaries. However, the
possible provision of such access and the possible positions of such boundaries
with respect to the layers of the IUT are factors to be taken into consideration in
the definition of the test methods, which may take advantage of this access  to
define the test suites in terms of the corresponding ASPs. It does  not  matter
whether the accessible boundaries are accessed via service access points (SAPs) or
via some other PCOs.

       Figure 8  shows  examples  of  IUTs,  with  respect  to  layer  boundary
accessibility.

Note - In addition, a conformance test suite Recommendation* may define "abstract
local primitives". These are used to specify control and observation of events or
states which are referred to in the protocol Recommendation* but which are internal
to the IUT and which cannot be expressed in terms of ASPs.  They are abbreviations
for text descriptions of control and observation to be performed by  the  upper
tester.

       Similar consideration  apply  to  relay  systems  (see  Part  2  of  the
Recommendation for details).











                            FIGURE 8/X.290, PART 1

                        Examples of IUT configurations

7.4    Abstract testing functions

       The definition of abstract  test  methods  requires  that  the  PCOs  be
distributed over two abstract testing functions, the lower and upper testers.

       The lower tester is the abstraction of the means of providing, during test
execution, control and observation at the appropriate PCO either below the IUT or
remote from the IUT, as defined by the chosen abstract test method. Thus, it is the
testing function related to the control and observation of the lower boundary of
the IUT. If the action of the tester is local to the SUT, the lower tester will
take the place of the lower part of the SUT. If the action  of  the  tester  is
external to the SUT, the lower tester will rely on the (N-1)-Service,  provided
jointly by the lower tester itself, a link and the SUT.

       The upper tester is the abstraction of the means of providing, during test
execution, control and observation of the upper service boundary of the IUT and of
any relevant abstract local primitive.

       There is a need for cooperation between the upper tester and  the  lower
tester; the rules for such cooperation are called the test coordination procedures.

       The test methods will vary  in  the  way  that  they  specify  the  test
coordination procedures. In some cases it is possible to define a test management
protocol to provide the coordination between the upper and lower testers. In other
cases it is only possible to describe the requirements to be met  by  the  test
coordination procedures without specifying what mechanisms might be used to realize
them.

7.5    Overview of abstract test methods

7.5.1  End-system IUTs

       For the IUTs defined within end-system SUTs (configurations 1 and  2  in
Figures 2 and 3) four categories of abstract test methods are defined, one local,
and three external ones which assume the lower tester is located remotely from the
SUT and connected to it by a link or network.

7.5.2  The local test methods

       The local abstract test methods define the PCOs as being at the  service
boundaries above and below the IUT. The test events are specified in terms of the
ASPs above the IUT and the ASPs and PDUs  below  the  IUT,  as  illustrated  in
Figure 9(a). Abstractly, a lower tester is considered to observe and control the
ASPs and PDUs below the IUT, while an upper tester observes and controls the ASPs
above the IUT. Requirements to be met by test coordination procedures  used  to
coordinate the realizations of the upper and lower testers are defined  in  the
abstract conformance test suites, although  the  test  coordination  procedures
themselves are not.







7.5.3  External test methods

       The external test methods use control and observation of the ASPs below the
IUT by means of a lower tester separated from the SUT, together with control and
observation of the ASPs above the IUT. Three different categories  of  external
abstract test methods are  defined,  which  are  referred  to  as  distributed,
coordinated, and remote. They vary according to the  level  of  requirement  or
standardization put on the test coordination procedures, on the access to the layer
boundary above the IUT, and on the requirements on an upper  tester.  They  are
illustrated in Figures 9(b), (c) and (d).

       The coordinated test method requires that the test coordination procedures
used to coordinate the realization of the upper and lower testers be achieved by
means of test management protocols. The other  two  methods  do  not  make  any
assumptions about the realization of the test coordination procedures.

       The distributed and coordinated test methods require specific  functions
from the upper tester above the IUT. The remote method does not.

       The distributed method requires access to the upper boundary of the IUT.
The other two methods do not.

7.5.4  Variants of end-system test methods

       Each category of test methods has variants which can be applied to single-
layer IUTs or to multi-layer IUTs. For multi-layer IUTs in which the protocols are
to be tested layer by layer, embedded variants can be used.

       All abstract test methods for end-systems are fully specified in section 8
of Part 2 of this Recommendation, including their single-layer, multi-layer and
embedded variants where applicable.

7.5.5  Relay-system IUTs

       For open relay-systems, two test  methods  are  defined,  loop-back  and
transverse. These  are  fully  specified  in  section  8  of  Part  2  of  this
Recommendation.             FIGURE 9/X.290, PART 1

                       Overview of abstract test methods
7.6    Applicability of test methods to real open systems

       The architecture and stage of development of a real open system determines
the applicability of test methods to it.

       Local test methods are useful for systems under development, when  their
architecture permits the isolation of an IUT, be it single-layer or multi- layer.

       External test methods are useful for testing complete  or  partial  end-
systems which can attach to telecommunications networks.

       Coordinated test methods apply where  it  is  possible  to  implement  a
standardized test management protocol in an upper tester in the SUT, above the IUT.












       Remote test methods apply when it is possible to make use of some functions
of the SUT to control the IUT during testing, instead of using a specific upper
tester.

       Distributed test methods apply when it is necessary  to  allow  complete
freedom for the implementation of the test coordination procedures between the SUT
and the lower tester, but to specify in  detail  the  control  and  observation
requirements at both ends.

       Single-layer test methods are the most appropriate methods for testing the
majority of the protocol conformance requirements.

       Multi-layer test methods will be used when conformance to true multi- layer
dynamic conformance requirements is to be tested.

       Embedded test methods permit the application of single-layer testing to all
layers of a multi-layer IUT.

       For 7-layer open systems, the preferred methods are the incremental use of
appropriate external single-layer embedded methods with the following PCOs:

       a)   the upper interface of the application layer as provided by the 7-
            layer open system, when applicable;

       b)   successively, each SAP (or corresponding PCO if there is no SAP as
            such) below the protocol which is the focus of  the  testing,  as
            controlled and observed in the external lower tester, starting from
            the lowest protocol of the IUT and working upwards.

7.7    Applicability of the test methods to OSI* protocols and layers

       The test methods defined in this Recommendation apply to all layers, with
the exception of the Physical-layer and the Media Access Control protocols which
are outside the scope of this Recommendation. Appendix I of this Recommendation
provides guidance on the applicability of test methods to all other layers.

8.     Test suites

8.1    Structure

       Test suites have a hierarchical structure (see Figure 10) in which  an
important level is the test case. Each test case has a narrowly defined purpose,
such as that of verifying that the IUT  has  a  certain  required  capability
(e.g. the ability to support certain packet sizes) or exhibit a certain required
behaviour (e.g. behave as required when a particular event occurs in a particular
state).

       Within a test suite, nested test groups are used to provide a  logical
ordering of the test cases. Test groups may be nested to an arbitrary depth. They
may be used to aid planning, development, understanding or execution of the test
suite.

       Test cases are modularized by using named subdivisions called test steps.
Each test case comprises at least one test step: the ordering of events covered
by the test purpose ("test body"). It may include further test steps to put the
IUT into the state required for the test body to start from (the "preamble") or
return to a quiescent state after the test body has finished (the "postamble").







       For practical reasons, common test steps may be grouped together  into
test step libraries. Test step libraries may be structured into nested sets of
test steps, to any depth of nesting. Test step libraries may be associated with
the whole test suite or with a particular test group or test case.

       Furthermore, all test steps consist of an ordering of other test steps
and/or test events (e.g. the transfer of a single PDU or ASP to or from the IUT).
All test steps are, therefore, equivalent to an ordering of test events (after
expansion of the inner test steps).
                           FIGURE 10/X.290, PART 1

                            Test suite structure
8.2    Generic, abstract and executable test cases

8.2.1  A generic test case is one which:

       a)   provides a refinement of the test purpose;

       b)   specifies that all sequences of test events (paths) in the test body
            which correspond to verdicts of "pass", "fail" and "inconclusive",
            using a specialized notation;

       c)   is used as the common root of corresponding abstract test cases for
            different abstract test suites for the same protocol;

       d)   includes a description of the initial state in which the test body
            should start, in lieu of a preamble;

       e)   need not describe the postamble;

       f)   is specified using the style of either the Remote  or  Distributed
            Single-layer test methods.

8.2.2  An abstract test case is derived from a generic case and  the  relevant
protocol specification; it:

       a)   specifies the test case in terms of a particular test method;

       b)   adds a more precise specification for sequences of events which are
            only described informally in the generic test case;

       c)   adds the sequences of test events required to  achieve  the
            objectives of the preamble and postamble of the generic test
            case using a specialized notation.

8.2.3  An executable test case is derived from an abstract test case, and is in a
form which allows it to be run on a real tester for testing a real implementation.

8.2.4  The terms generic, abstract and executable are used  to  describe  test
suites, which comprise generic, abstract and executable test cases, respectively.

8.2.5  A generic test suite has the coverage of the set or a superset  of  all
possible abstract test suites for a particular protocol.












9.     Relationships between concepts and roles

       Figure 11 is a pictorial representation of  the  relation  between  the
various Recommendations and the processes of producing generic,  abstract  and
executable test suites and test reports.

       Part 2 concerns the production of testable protocol Recommendations and
abstract test suite Recommendations. Part  1  provides  general  concepts  and
definitions.

Note - Other aspects of the conformance assessment process, such as executable
test derivation, preparation of the IUT, PICS and PIXIT by the client and test
laboratory role are for further study.

10.    Compliance

       In this Recommendation, "compliance" refers to meeting the requirements
specified by the Recommendation. This word is used as an attempt to  eliminate
confusion between compliance to this Recommendation and conformance of a protocol
implementation to protocol Recommendations.

       This part of the Recommendation contains no compliance requirements.
                           FIGURE 11/X.290, PART 1

                   Relationships between concepts and roles